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In this brief, we present a method for predicting the impact of school closures on 
average ACT® test scores. We use ACT test-retest data to estimate the average 
per-month gain in ACT scores while schools are in session and the average per- 
month loss in ACT scores while schools are not in session. We then apply the 
estimates to predict the impact of school closures on average ACT test scores 
under three different scenarios for school closure, school re-opening, duration 
and effectiveness of online instruction, and ACT test dates. 


The purpose of this brief is to illustrate a methodology and major conditions that 
should be considered in predicting aggregate ACT scores when school closes 
prematurely and learning and testing are disrupted. Across three scenarios 
examined below, the impact on academic achievement was predicted to be 
between -0.71 and -0.31 Composite score points. The predictions are based on a 
number of assumptions and may not be applicable in every case. ACT can assist 
our state and district partners in applying and interpreting this or similar models 
to predict the effect of COVID-19 on learning and performance within their local 
context. 


Estimating Per-Month Gains in ACT Scores 


Using data from the 2017-2018 and 2018-2019 ACT testing years, we generated 
a data set of students’ first two ACT tests. To be included, students must have 
attended a high school in the United States, been in grade 11 when they first 
took the ACT, and have taken the ACT at least twice. This resulted in a sample of 
482,413 students. 


We identified each high school’s usual start and end date using a database of 
high schools. We calculated the number of days between the first and second 
ACT test that the student was in school and the number of days between tests 
that the student was not in school. 


As an example, suppose a student first took the ACT on April 14, 2018 and took 
the ACT again on October 27, 2018. Further, suppose their school closed on 
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May 25, 2018 and re-opened on August 20, 2018. In this case, there were 109 days 
between tests that the student was in school, and 87 days that the student was not 
in school.' 


Next, we used linear regression to estimate the gain in ACT scores associated 

with each additional day that school was in session, and the gain (or loss) in ACT 
scores associated with each additional day that school was not in session. Finally, 

we transformed the per-day gain estimates to per-month estimates (Table 1). While 
positive gains were observed while school was in session, losses were observed while 
school was not in session, evidence of summer learning loss. In-school gains were 
largest for English (0.310 points per month), and out-of-school losses were largest for 
math (-0.221 points per month). 


One way to interpret the Composite per-month in-school gain and out-of-school loss 
estimates is that students lose about two months of school learning every summer 
(assuming a three-month summer break). This is larger than the loss of one month of 
school learning typically observed and may be due to the greater extent of summer 
learning loss for upper grade levels (Cooper, Nye, Charlton, Lindsay, & Greathouse, 
1996). Our results suggest that summer loss is more detrimental for math than for 
reading, consistent with prior studies (Cooper et al., 1996; Kuhfeld & Tarasawa, 2020). 


Table 1. Per-Month ACT Score Gains 


Per-Month Gain 


ACT Score Ta bscxeq alee) | OJ T] oy Yor pYore) | 
Composite 0.218 -0.142 
English 0.310 -0.159 
Mathematics 0.193 -0.221 
Reading 0.182 -0.023 
Science 0.186 -0.164 


From Table 1, we can make some general predictions, such as: 


e Under normal circumstances, ACT Composite score increases by 1.96 score 
points during the typical nine-month academic year, and then decreases by 0.43 
score points over a three-month summer break, resulting in a net gain of 1.53 
score points over an entire year. 


e Assuming schools shut down for two months due to COVID-19 and that learning 
loss is the same during the shutdown as it us during normal summers, student 
achievement would drop by 0.72? ACT Composite score points, resulting in a net 
gain of only 0.82° score points over an entire year. 


For most students, 0.72 Composite score points may not seem consequential because 
it means they are only slightly less prepared for college and career, and this would be 
unlikely to impact admissions and scholarship decisions. But for states and districts, 
0.72 Composite score points can cause a significant disruption in trend analyses 

and could be considered a rather large and important effect on aggregate student 
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achievement. While average ACT scores can vary somewhat across cohorts, the 
COVID-19 school closures may become a new source of variation in score trends. 


Predicting the Impact of COVID-19 School Closures on 
Average ACT Test Scores 


We now walk through three examples showing how the per-month gain estimates from 
Table 1 can be used to predict the impact of COVID-19 school closures on mean ACT 
scores and student achievement. The impact on student achievement can be thought 
of as the net impact on the academic achievement of a cohort of students, regardless 
of when the students take the ACT test. The impact on mean ACT scores is similar to 
the impact on student achievement but is influenced by when the ACT test is taken. 
For example, the impact on mean ACT scores is reduced when students have more 
days in school before taking the ACT. 


In our examples, the predicted impacts depend on: 


e Date of school shutdown 

e Effectiveness and duration of online instruction during shutdown 
e Planned ACT test date for spring grade 11 

e Usual school end date 

e School start date for fall 2020 

e ACT makeup test date, if any 


Predicting the impact of school closures on mean ACT scores and student achievement 
requires strong assumptions, including: 


e Average per-month ACT score gains while schools are in session are the same 
as that observed in our study. 


e Average per-month ACT score losses while schools are not in session are the 
same as that observed in our study. 


e The rates of gains and losses are linear and the same across different months. 


We must also make assumptions about average per-month gains or losses during the 
shutdown. One could assume that the losses during the shutdown would be the same 
as those observed under normal circumstances when schools are not in session. Or, 
with mitigation efforts such as online learning, one might assume that gains during 

the shutdown are some percentage of the gains that would be expected under normal 
circumstances when schools are in session. In addition, online learning may have been 
used for some fraction (e.g., 50%, 80%) of the days remaining on the original school 
calendar. 


Predicting the impacts of COVID-19 closures on mean ACT scores does not 
necessarily require a prediction of what the mean ACT score would have been, but 


Mean ACT Composite 
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rather, it only requires an estimate of the difference in the predicted mean score under 
normal circumstances and what is predicted after the COVID-19 closures.* 


Example Scenario 1: Fall 2020 Makeup Test with No 
Online Instruction and Early Start to the 2020 School Year 


Assumptions used for the example are presented in Figure 1 and include: 


e No online instruction during shutdown 
e March 13 school shutdown 
e April 21 planned ACT test date for 11th graders 
e May 27 usual school end date 
e August 10 early school start date for fall 2020 
e August 25 usual school start date 
e October 1 makeup ACT test date 
Figure 1 shows the trajectories for predicted mean ACT Composite scores under 


normal circumstances (solid blue line) and under COVID-19 shutdown with no online 
learning (dashed orange line). 


Figure 1. Examples 1 and 2 for Predicting Impacts of COVID-19 Shutdowns 
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Predictions Under Normal Circumstances 


At the time of the shutdown (March 13), the predicted mean ACT Composite score 
was 20.42. This is calculated using the predicted mean ACT Composite score at the 
beginning of the school year (August 27, 19.0) and adding 0.218 score points per 
month through March 13 (approximately 6.5 months). Without the shutdown, the 
predicted mean ACT Composite score would have continued to increase by 0.218 
points per month, landing at 20.70 for the planned ACT test date of April 21. With 
continued gains through the end of the school year (May 27), the predicted mean rises 
to 20.96. With summer learning loss occurring from May 28 through August 24, the 
predicted mean drops to 20.54 at the time that school would normally resume. Gains 
resume from August 25 through October 1, at which time the predicted mean is 20.82. 


Predictions Under COVID-19 School Closures with No Online 
Learning Supplement 


Because of the shutdown, the predicted mean ACT Composite score begins to decline 
March 13 and continues to decline by 0.142 score points per month through August 

9, landing at 19.73. If school resumes early on August 10, the predicted mean ACT 
Composite score increases from August 10 to October 1, landing at 20.11 for the 
October 1 test date. 


In this scenario, the impact of the COVID-19 shutdown on academic achievement 
is 0.71 ACT Composite score points. This is the difference in predicted mean ACT 
Composite score on October 1 under normal circumstances (20.82) and with the 
COVID-19 shutdown (20.11). 


Also under this scenario, the expected impact on average Composite score is -0.59 
points—the difference between the predicted mean score on the makeup test date due 
to the shutdown (20.11 on October 1) versus that on the original test date under normal 
circumstances (20.70 on April 21). The COVID-19 impact on average Composite score 
is represented by the vertical difference between the blue and orange stars shown in 
Figure 1. 


Example Scenario 2: Fall 2020 Makeup Test with 50% 
Effective Online Instruction and Early Start to the 2020 
School Year 


In the next example, we assume that students receive online instruction that results 

in 50% of the gains of regular instruction,° that the online instruction begins April 10— 
about four weeks after schools are closed—and continues through May 27. Otherwise, 
the assumptions are the same as those made in Example 1. Figure 1 (dashed 
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green line) shows the trajectories for predicted mean ACT Composite score under 
this scenario. 


Predictions Under COVID-19 School Closures with 50% 
Effective Online Learning Supplement 


Instead of the predicted mean ACT Composite decreasing by 0.142 points per month 
from April 10 (online instruction begins) to May 27, it increases by 0.109 (50% of 0.218) 
points per month. With 48 days (1.58 months) during this period, this results in a net 
gain of 0.40 score points due to the online instruction.® 


Therefore, instead of the COVID-19 shutdown having an impact on academic 
achievement of -0.71 Composite score points, the impact is only -0.31 score points. 
Further, instead of having an impact of -0.59 points on average ACT Composite score 
as it had under Example 1, the shutdown has an impact of -0.19 points for Example 2. 
Of course, we cannot empirically determine the learning gain of online instruction since 
COVID-19 is unprecedented and we do not have historical data to use in estimating 
the effects of online learning under such conditions. We used 50% for this example, 
but one could substitute any value as an estimate of the percent of instructional gain 
associated with online learning compared to in school learning (e.g., 10%, 50%, 100%). 
It is also possible that learning loss occurs with online instruction but is less severe 
than what happens during normal summers. In this case, one would assume a per- 
month loss that is some percentage of the normal loss. 


Example Scenario 3: No Makeup for ACT Test, but What 
Would Average Score Have Been? 


We now walk through a final example of how the per-month gain estimates from 

Table 1 can be used to predict the impact of COVID-19 school closures on mean ACT 
scores, as well as what the mean ACT Composite score would have been if testing had 
occurred on the planned ACT test date. This scenario assumes no makeup for ACT 
testing but rather predicts what the average score for a state would have been for their 
cohort of students. The assumptions used for the example include: 


e March 13 school shutdown 


e April 21, 2020 planned ACT test date for 11th graders 


e No online instruction between the beginning of shutdown to the planned ACT test 
date 


e No makeup ACT test date 


e The mean ACT Composite score for state-tested 11th graders over the past three 
years is 19.50, with an average test date of April 21 


Because the state has tested around April 21 for the past three years, we can use 
the average ACT Composite score over the past three years (19.50) as a starting 
point, assuming that the current cohort of students is similar to prior cohorts on other 
measures of academic achievement. There are 39 days (1.28 months) between the 
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shutdown date and the planned ACT test date. Instead of gaining 0.218 score points 
per month, we can assume a loss of 0.142 score points per month. Therefore, the 
prediction of the COVID-19 impact is -0.46 score points (1.28*(-0.142-0.218)), and the 
predicted mean ACT Composite score is 19.04. 


Conclusions and Cautions 


We have attempted to provide a simple model to predict aggregate ACT scores for a 
large school, district, or state that closed during the school year and was unable to test 
all juniors during their typical spring semester testing window. The examples illustrate 
how to answer the following questions for the school, district, or state under these 
conditions: 


e What is the impact of school closures on mean student achievement, as 
measured by the ACT test? 


e Ifamakeup ACT test is administered, how much impact will school closures have 
on average ACT scores? 


e What would have been the average ACT scores for a cohort of students if testing 
occurred on the regular schedule? 


The three examples use the same model but under slightly different conditions. 
Example 1 predicts the ACT score when schools close prematurely, annual testing 

is delayed until the following fall, school starts early the next fall, and there is no 
instructional gain assumed during the school closure (or no instruction). Example 2 

is similar but assumed that online instruction began four weeks following closure and 
had 50% of the instructional gain associated with in-school learning under normal 
circumstances. Example 3 assumes no online instruction, no makeup testing is 
conducted, and that a state is interested in predicting what the mean ACT score would 
have been if testing was completed on the originally scheduled date. A variation of 
Example 3 could include online instruction, but additional testing assumptions must be 
met if retesting is never conducted making the predictions less accurate.’ 


In summary, these are very coarse methods used to predict aggregate ACT scores, 
which require a number of assumptions. The predictions are likely to be more accurate 
for large cohorts (>100 schools, >10,000 students) that are similar to prior cohorts of 
students. We avoided using other factors that could have contributed to the models 
(e.g., demographics of the cohort, school/district/state size and context) because they 
would have made the model more complex (less transparent) and imposed a false 
sense of precision. 


As noted earlier, ACT is unable to estimate learning gains associated with online 
learning alone compared to in-school learning, and the decision on how best to weight 
this proportion (online to in-school learning) could be set at any value using data a 
school may possess or by making rational assumptions. Another way to adjust online 
learning may be by participation rate. For example, if only 65% of students in the 
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cohort are actively engaged and attending most online instructional opportunities, 
a second adjustment could be included in the prediction reflecting this participation 
rate. 


Notes 


‘2 


Note that the number of days in school is not the same as the number of 
instructional days because weekends and holidays are not counted as 
instructional days. 


This is calculated by multiplying the number of months (2) times the expected 
loss per month less the expected gain that would have occurred under normal 
circumstances (-0.142—0.218). 


ACT Composite score would be predicted to increase by 1.52 score points 
during the shortened seven-month academic year, and then decrease by 0.71 
for the extended out-of-school duration of five months. 


For instance, in Examples 1 and 2, the predicted mean ACT Composite 

score is 19.0 at the beginning of the 2019-2020 academic year for illustrative 
purposes. But the prediction of the COVID-19 impact does not depend on this 
starting value and so would not change with different starting values (one of 
the simplifying assumptions of this approach is that per-month gain or loss 
does not depend on a school’s starting value for mean ACT Composite score). 
In Example 3, the goal is to predict what the mean ACT Composite score 
would have been if the tests were not cancelled, in addition to predicting the 
impact of COVID-19 on student achievement. 


Any value from 0-100% could be substituted here; 50% was used for 
illustrative purposes to assume a lower level of effectiveness of online 
instruction. 


This is calculated as ((0.109*1.58) - (-0.142*1.58)) = 0.251*1.58 = 0.40. 


For example, if there is no ACT makeup test, we must base the predicted ACT 
scores on an additional assumption, such as the students having the same 
achievement (under normal circumstances) as the state’s average for the 
past three cohorts. This assumption may be problematic for many schools, 
districts, and states if participation rates or population changes occurred, or 

if instructional interventions and other factors may have had an impact on 
learning and performance. 
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